# CVE Scan

The osquery project has a CI job which once a day scans for CVEs that are present and not yet addressed in its third party libraries.
The scan is done by a python script at `tools/ci/scripts/cve/third_party_libraries_cves_scanner.py`, which uses the NIST database queried via their NVD APIs; a manifest file at `libraries/third_party_libraries_manifest.json` contains the list of third party libraries and their metadata necessary to correctly download the CVEs.

The manifest file format is validated everytime the CVEs download happen and on every PR by the script `tools/ci/scripts/cve/validate_maninfest_libraries_scanner.py`; additionally the third party library versions in the manifest are verified, to ensure that they are up to date with their state in the repository.

After having downloaded the list of CVEs, the script will open issues in the osquery repository, for all the unresolved CVEs, checking against the already opened ones to prevent duplicates, and it will go back in time up to 6 months old.  
The issues can be recognized because they will be opened by the `github-bot` author and will have the `security`, `libraries`, `cve` and `severity-<level>` labels on them.

NOTE: This product uses the NVD API but is not endorsed or certified by the NVD.

# Updating a third party library to resolve a CVE

The process of updating a third party library is the usual, but in the PR updating the library the contributor MUST:

   1. Link the CVE(s) issue(s) the PR is going to close, so that when it's merged, they are automatically closed
   2. Update the manifest and specifically the `version` and `commit` fields with the information of the new library.
      Remember that the `commit` has to be the one of the submodule in the osquery repository, which might not always match the commit of the library original repository.

Failing to do step 1. only leads to having to manually close those issue and link them back to the PR for tracking purposes.

Failing to do step 2. will lead to the PR not being mergeable because the CI checks the `commit` field against the actual git submodule commit.
Note that if the `commit` is updated but not the `version`, this will not be detected by the CI and the periodic scan will use the incorrect
version to download CVEs, finding again the fixed CVE and reopening the issue.
If this happens, one just needs to do another PR that updates the `version` correctly.

Any other situation where the `version` is incorrect (older than what previously was or newer than what actually is) is still not detected,
and will either cause the script to open issues for already fixed CVEs or to miss CVEs, so it's very important that the PR review process
double checks the new `version`.

Important: Do not merge this kind of PR if the CI CVE scan job is running (which happens only once a day between 23:00 and 00:00 UTC), otherwise the job could start with an old view of the repository and open new issues on already fixed CVEs.
If this happens, we just need to close or even delete those issues, but it's mostly to avoid additional work or confusions.

# Ignoring a CVE not affecting osquery

There are cases where the API returns CVEs that are not affecting a third party library, not directly, but they are affecting other software that uses the third party library. There might be something we can do in the future to resolve what seems a bug in the API, but for now in the manifest it's possible to list CVEs that should be ignored, so that issues for those are not opened again in the future.

Additionally we often have the case where a CVE is not affecting osquery due to how or what parts of the third party library are used, so having a way to ignore a CVE helps with that too.

The process therefore is to:
   
   1. Open a PR which updates the manifest and specifically updates the `ignored-cves` field of the library the CVE comes from.
   2. Describe in the CVE(s) issue(s) the reason why they are going to be closed
   3. Link the above issues to the PR, so that they are closed when the PR gets merged

Important: As with updating a library, one has to ensure that the CI CVE scan job is not running

# Adding a new library

When a new library gets added, the manifest needs to be updated too, otherwise the CI check that verifies the manifest in the PR will fail.

Currently the JSON format for a third party library as a submodule (taking as an example `libdpkg`) is:

```json
"libdpkg": {
    "product": "dpkg",
    "vendor": "debian",
    "version": "1.21.7",
    "commit": "e61f582015a9c67bbb3791cb93a864cfeb9c7151",
    "ignored-cves": []
},
```

The name of the library, `libdpkg`, and the `commit` field must match the name of the folder containing the submodule source code folder, and the commit at which the submodule currently is, respectively.

The `product`, `vendor` and `version` fields are used in the NVD APIs instead, and they must match what the NIST database uses.  
This is a matter of using the CPE search at https://nvd.nist.gov/products/cpe/search, and trying to find which are the correct `product` and `vendor`
using a cpe like `cpe:2.3:a:*<partial vendor guess>*:*:*:*:*:*:*:*:*:*` and for the product `cpe:2.3:a:<vendor>:*<partial product guess>*:*:*:*:*:*:*:*:*`.

Another way could be to download the full dictionary from https://nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.3.xml.gz, then playing with grep and awk.  
For instance if we want to see all the unique products that the `amazon` vendor has:
```sh
cat official-cpe-dictionary_v2.3.xml | grep -Eo "cpe:2.3:a:amazon:[^\"]*" | awk -F ":" '{ print $5 }' | sort | uniq | less
```

For the `ignored-cves` refer to [Ignoring a CVE not affecting osquery](#ignoring-a-cve-not-affecting-osquery); the field will likely be empty at the beginning.

## Special cases

### Libraries without a CPE

Some libraries do not have a CPE assigned, so no CVEs will be found in the NIST database. We track these libraries in the manifest, because the validation script will check for it, but one can provide less fields; only `vendor` and `commit` are required.

The script though also needs to know that this is a library of that kind, so the contributor has to update the list of libraries that does not have a CPE in `tools/scripts/ci/cve/osquery/manifest_api.py`

### Libraries not imported as a submodule

Right now there's only one case and ideally, the only, but `openssl` is not a submodule, so there's no `commit` to use to check if the manifest is up to date. `version` is used instead, and it's parsed from the CMake file at `libraries/cmake/formula/openssl/CMakeLists.txt`.  
If it will ever happen that osquery needs to add another library of this kind, then logic to get its version should be written in the `tools/scripts/ci/cve/validate_manifest_libraries_versions.py` script.

Additionally the name of the library should be added in `tools/scripts/ci/cve/osquery/manifest_api.py` and finally the manifest fields requirements would be the same as for a normal library, just without the `commit` field.

### Libraries not used in the release build

Another case is when the library is only used for testing purposes; this doesn't need to be tracked in the manifest, but still needs to be ignored by the script that checks that all the necessary libraries are present and up to date in the manifest.
The script to update is `tools/scripts/ci/cve/validate_manifest_libraries_versions.py` (the current example is `googletest`).
